Method of and microscope comprising a device for detecting movements of a sample with respect to an objective

ABSTRACT

For detecting movements of a sample with respect to an objective, the sample is imaged onto an image sensor comprising an array of pixels by means of the objective. Images of the sample are recorded in that light coming from the sample is registered at the pixels. Variations of intensities of the light coming from the sample and registered at the pixels are determined during a set-up period in that a temporal course of the intensity of the light, which has been registered at a respective one of the pixels over the set-up period, is analyzed. Using these variations as a criterion, a subset of not more than 90% of the pixels of the image sensor is selected. Parts of the images that each correspond to the selected subset are compared to parts of at least one reference image that also correspond to the subset.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to German Patent Application No. DE 102020 127 071.2 filed Oct. 14, 2020.

FIELD OF THE INVENTION

The present invention relates to a method of detecting movements of asample with respect to an objective.

Further, the invention relates to a microscope comprising an objectiveand a device for detecting movements of a sample with respect to theobjective.

Movements of a sample with respect to an objective of the microscope bymeans of which the sample is examined have a significant effect. Thisespecially applies in super-resolution microscopy in which spatialresolutions beyond the diffraction barrier are achieved, and even if themovements do not exceed the nanometer range. Any movement of the samplewith respect to the objective, which takes place between two points intime, shifts the relative positions of objects in the sample which havebeen determined at these two points in time. If movements of the samplewith respect to the objective are not detected, they cannot becompensated, and the effective spatial resolution in microscopicallyimaging a structure of interest is determined by the extent of thesemovements.

Particularly over longer measurement periods, movements of a sample withrespect to the objective of a microscope may never be avoidedcompletely, and limiting these movements to small values also incursconsiderably technological effort. Thus, there is an interest indetecting these movements to be able to consider and, particularly, tocompensate them.

BACKGROUND OF THE INVENTION

For detecting movements of a sample with respect to an objective, it isknown from international patent application publication WO 2020/201 430A1 to image light from at least one reference object connected to thesample at consecutive points in time by means of the objective intoimages in an image plane. The images in the image plane are recorded bya camera which is used as an image sensor, and they are compared toreference images. Low spatial frequencies are masked out of the imagesin a plane that is Fourier-conjugated to the image plane in front of thecamera. For this purpose, parts of the light, which originate from acentral area of a pupil of the objective, are masked out in the planethat is Fourier-conjugated to the image plane. The reference objectsshall have edges over which the intensity of the light originating fromthe reference objects drops by at least 90% and whose width in parallelto the image plane is smaller than the wavelength of the light. Thereference objects may be point shaped markers or beads having a diameterbelow the wavelength of the light. The reference objects may also besuitable structures of the actual sample. A lateral movement of thesample with respect to the objective, that is orthogonal to an opticalaxis of the objective, is gathered from shifts of object images of thereference objects between the images. On the other hand, an axialmovement of the sample with respect to the objective, that is orientedalong the optical axis, is gathered from deformations of the objectimages of the reference objects and particularly from similarities ofobject images of the reference objects in the images with object imagesof the reference objects in the reference images. It does not appearfrom WO 2020/201 430 A1 how the object images of the reference objects,especially of suitable structures of the sample itself, may be found inor even be automatically selected from the images.

From the “Supplementary Materials” to F. Balzarotti et al., “Nanometerresolution imaging and tracking of fluorescent molecules with minimalphoton fluxes”, Science, Vol. 355, Issue 6325, pages 606-612, 2017, itis known to detect the axial position of a sample with respect to anobjective by means of the movement of the beam image of a reference beamirradiated at an angle and totally reflected at a cover slip boundarysurface in an image of the cover slip recorded with a camera. A lateralsample position with respect to the objective is detected by imaging adark field image of scattering nanorods in the sample onto a furthercamera. Two-dimensional Gaussian functions are fitted to the nanorodimages of the nanorods, and the center of the respective function isused as a measure of the lateral position of the respective nanorod. Inthis document it is also not described how the nanorod images of thenanorods can be found in or even be automatically selected from the darkfield images.

K. C. Gwosch et al., “M INFLUX nanoscopy delivers multicolor nanometer3D-resolution in (living) cells”, bioRxiv, doi:http://dx.doi.org/10.1101/734251, 2019, in “Materials and Methods”,disclose an active stabilization system for position stabilization of asample with respect to an objective. For lateral stabilization,scattering gold nanorods are imaged onto a camera. The axial position isdetected by illuminating the sample with an infrared laser beam totallyreflected at the sample. Once again, there is no indication how thenanorod images of the individual gold nanorods may be automaticallyfound in the images of the camera or even be automatically selectedtherefrom.

A method of operating a microscope and a control unit for a microscopefor realizing an autofocus with angle-variable illumination are knownfrom German patent application publication DE 10 2018 107 356 A1 and USpatent application publication US 2019/0 302 440 A1 belonging to thesame patent family. At least one image is captured in a multiplicity ofangle-variable illumination geometries. A separation of an object imageof a measurement object from disturbing structures in the at least oneimage is carried out on the basis of control data indicative of a prioriknowledge. After the separation, components in the at least one imagethat change in relation to a change in the angle-variable illuminationgeometry are recognized as an object shift of the measurements object.Based on the object shift, a defocus position of the measurement objectis determined and then compensated by adjusting a z-position of a samplestage of the microscope. The disturbing structures may, for example, belight reflections, shades, effects due to impurities, for example in thearea of the sample stage, but also in static regions of an imaging opticof the microscope, and sensor noise of the detector. For identifying thedisturbing structures, the measurement object may be moved inz-direction. Spatially fixed disturbing structures which are notconnected to the sample stage are then spatially fixed and can beacknowledged by calculating differences. Generally,reference-measurements will be used, in which an image withoutmeasurement object or variable measurement objects are captured, forexample, in a calibration phase prior to the actual measurement. Then,in a corresponding reference image, the disturbing structures which arecaused by the imaging optic of the optical system are detected. Inanother implementation, the contrast is considered in combination with asignal to noise ratio. For this purpose, a pair-wise correlation betweenimages of a plurality of images is calculated, and a correlation maximumis detected in each correlation. Then, it is requested that thecorrelation maximum does not go below or exceed a predetermined limitvalue. The limit value may, for example, be determined based on thecontrast of the disturbing structures. For low contrast measurementobjects, those correlation maxima with particularly high values may bediscarded.

An apparatus for taking images and a method of taking images withreflection suppression are known from German patent applicationpublication DE 10 2014 113 256 A1. An object is illuminated under aplurality of illumination geometries. A detector registers a pluralityof images of the object for the plurality of illumination geometries. Anelectronic evaluation device applies a shadowing operation forreflection suppression to at least a part of the plurality of theimages. The shadowing operation for reflection suppression depends onthe illumination geometry utilized in recording the respective image.The modified images generated by the shadowing operation are combinedinto a resulting image.

A method of reducing image artifacts in images is known from Germanpatent application publication DE 10 2017 125 799 A1 and US patentapplication publication US 2020/0 265 570 A1 belonging to the samepatent family. Images are captured at different arrangements of a sampleobject to an illumination and a detector. Then, based on a comparison ofpixel values of pixels of the images, an pixel-wise combination of theimages takes place. In this way, an artifact reduction is achieved, i.e.reflections and/or shading-in can be reduced.

A method of digitizing microscopic images of a biological tissue isknown from US 2002/0 090 127 A1. At first, an image is converted into agreyscale image. Then, an average value and the standard deviation ofthe local pixel intensities are analyzed. The average pixel intensitiesare used to differentiate between regions containing tissue and emptyregions and other non-tissue regions of the image. The standarddeviations are a good indication of the limit between tissue and emptyimage. The average and the standard deviations are combined to generatea limit value which is used to carry out a preliminary classification oftissue with respect to non-tissue. Afterwards, morphological filters canbe applied to refine the classification based on the size and theposition of neighboring groups of potential tissue pixels.

From U.S. Pat. No. 9,068,944 it is known to reduce the size of lightintensity data in a scanning molecule counting method that isimplemented using a confocal or multiphoton microscope. A time series ofa light intensity of light from a detection area which is moved withrespect to the sample is analyzed to detect the signal of a lightemitting particle in the time series. Areas in which there is no signalthat indicating light from light emitting particles are removed from thelight intensity data of the light intensity time series.

There still is a need of a method of and a microscope comprising adevice for detecting movements of a sample with respect to an objective,in which the movements of the sample with respect to the objective canbe detected automatically, particularly without manual selection ofobject images of reference objects of the sample.

SUMMARY OF THE INVENTION

The present invention relates to a method of detecting movements of asample with respect to an objective. The method comprises the step ofimaging the sample onto an image sensor which comprises an array ofpixels by means of the objective; the step of recording images of thesample by the image sensor in that light coming from the sample isregistered at the pixels of the image sensor; and the step ofdetermining variations of intensities of the light coming from thesample and registered at the pixels of the image sensor during a set-upperiod in that a temporal course of the intensity of the light, whichhas been registered at a respective one of the pixels of the imagesensor over the set-up period, is analyzed. The method further comprisesthe step of selecting a subset of not more than 90% of the pixels of theimage sensor using the variations as a criterion; and the step ofcomparing parts of the images that each correspond to the selectedsubset of the pixels of the image sensor to parts of at least onereference image that also correspond to the subset of the pixels of theimage sensor.

The present invention also relates to a microscope comprising anobjective, a sample holder for positioning a sample relative to theobjective, and a device for detecting movements of the sample withrespect to the objective. The device comprises an image sensor includingan array of pixels, onto which the sample is imaged by means of theobjective, the image sensor being configured for recording images of thesample in that light coming from the sample is registered at the pixelsof the image sensor, and a selection module configured to determinevariations of intensities of light coming from the sample and registeredat the individual pixels of the image sensor during a set-up period inthat a temporal course of the intensity of the light registered at therespective pixel of the image sensor over the set-up period is analyzed,and to use the variations as a criterion in selecting a subset of notmore than 90% of the pixels of the image sensor. The device furthercomprises a comparison module configured to compare parts of the imageswhich each correspond to the subset of the pixels of the image sensorwith parts of at least one reference image which also correspond to thesubset of the pixels of the image sensor.

Other features and advantages of the present invention will becomeapparent to those skilled in the art upon consideration of the followingdrawings and the detailed description. It is intended that all suchadditional features and advantages be included herein within the scopeof the present invention, as defined by the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention can be better understood with reference to the followingdrawings. The components in the drawings are not necessarily to scale,emphasis instead being placed upon clearly illustrating the principlesof the present invention. In the drawings, like reference numeralsdesignate corresponding parts throughout the several views.

FIG. 1 is a schematic depiction of a microscope of the presentdisclosure.

FIG. 2 is a flowchart of an embodiment of the method of the presentdisclosure.

FIG. 3A to FIG. 3E are graphs explaining how to make a firstpre-selection in the method of the present disclosure.

FIG. 4A to FIG. 4E are graphs explaining how to make a thirdpre-selection in the method of the present disclosure.

FIG. 5A, FIG. 5B and FIG. 5C are graphs explaining how to form an imagemask in the method of the present disclosure.

FIG. 6A, FIG. 6B and FIG. 6C are graphs showing details of the thirdpre-selection in the method of the present disclosure, and

FIG. 7A, FIG. 7B and FIG. 7C show details of the first pre-selection inthe method of the present disclosure.

DETAILED DESCRIPTION

In a method of detecting movements of a sample with respect to anobjective according to the present disclosure, the sample is imaged ontoan image sensor by means of the objective. Images of the sample arerecorded by the image sensor in that light coming from the sample isregistered at pixels of the image sensor. A subset of not more than 90%of the pixels of the image sensor is selected. For this purpose,variations of intensities of the light coming from the sample andregistered at the individual pixels of the image sensor during a set-upperiod are determined, and the variations are used as a criterion inselecting the subset of the pixels of the image sensor. Afterwards, theparts of the images, which each correspond to the selected subset of thepixels of the image sensor are compared to a part of at least onereference image which also corresponds to the selected subset of thepixels of the image sensor. The reference images may be previouslyrecorded images of the sample in which the sample has been in a knownrelative position with respect to the objective.

Despite the determination of the variations and the use of thevariations as a criterion in selecting the subsets of the pixels of theimage sensor, the method according to the present disclosure maycorrespond to the method known from WO 2020/201 430 A1 or the methodsfor detecting lateral movements of a sample with respect to an objectiveknown from F. Balzarotti et al. and K. C. Gwosch et al.

For selecting the object images of suitable reference objects of thesamples in the recorded pictures of the sample, the variations of lightintensities of the light coming from the sample and registered by theimage sensor, which result at the individual pixels of the image sensorduring a set-up period, are determined. Thus, the temporal course of theintensity of the light registered at the individual pixels of the imagesensor is analyzed over the set-up period. For this purpose, theintensity of the light or of a signal of the image sensor belonging tothe respective pixel of the image sensor may be sampled at a suitablesample rate, or the intensity or the signal of the image sensor isintegrated over consecutive short intervals. The sample rate or theintervals are to be selected such that typically between 7 and 100single values are available as a base for determining each variation ateach pixel of the image sensor.

Actually, the variation for the respective pixel of the image sensor maybe determined as a standard deviation of these single values from theaverage value of the single values. In other words, the variations maybe determined as standard deviations of the intensities from an averageintensity of the light registered at the respective pixel of the imagesensor. A determination of the variation as a variance of theintensities of the light registered at the respective pixel of the imagesensor is also possible and generally of equal value, because thevariance and the standard deviation are in a fixed relation. Thevariances are the squares of the standard deviations. Considering thisfact, all the following explanations with regard to standard deviationsare transferable to variances of the intensities.

Considerable parts of the variations of the light intensities determinedin this way are caused by a background noise at the pixels of the imagesensor and statistically, particularly if only few photons of the lightcoming from the sample are the basis of the determination of therespective variation. Thus, it is generally suitable to correct thevariations for a variation background value which is linearly dependenton a square root of an average intensity of the light registered at therespective pixel of the image sensor, and with regard to the backgroundnoise, as long as the background noise at the pixels of the image sensordoes not remain so small as compared to the variation background valuethat it may simply be neglected. The correction for the background noisemay be made by subtracting a constant background noise value. Thecorrection for the variation background value may be made in that thevariation background value at the respective pixel is subtracted fromeach variation. Alternatively, the variations may be normalized in thateach variation is divided by the variation background value at therespective pixel.

The pixels of the image sensor of the highest variations, particularlythe highest corrected variations, are of high importance in selectingthe pixels of the image sensor which correspond to suitable referenceobjects of the sample for detecting movements of the sample with respectto the objective. Depending on the conditions during the set-up period,the height of the variations is to be considered as a positive ornegative criterion in selecting the pixels of the subset.

If the sample is set in motion with respect to the objective in theset-up period, the resulting variations are a suitable positivecriterion in selecting the subset of the pixels of the image sensor. Thevariations then point to object images of reference objects which, dueto the motion of the sample with respect to the object, move in aneasily determinable way so that the motion can be tracked by means ofeasily determinable changes of the images. On the other hand, pixels atwhich, despite the motion of the sample with respect to the objective,no remarkable variations occur are without information content fortracking these movements.

In an embodiment, the sample may be set in a first motion with respectto the objective in a first partial period of the set-up period.Variations resulting from the first motion over the first partial periodcan be determined and afterwards be used as a positive criterion inselecting the subset of the pixels of the image sensor. The selectionmay be implemented in that the first variations or the corrected firstvariations exceed a first variation limit value at all pixels of thesubset or at least within a predetermined maximum first distance to allpixels of the subset.

In a more particular embodiment, a first pre-selection for the subset ofthe pixels may be made in that all pixels are selected in which thefirst variations or the corrected first variations exceed the firstvariation limit value, and in that all pixels are added which arelocated within the predetermined maximum first distance to the pixels atwhich the first variations or the corrected first variations exceed thefirst variation limit value.

If here, in the following parts of the description or in the claims,reference is made to any pixels that are located within a predeterminedmaximum distance to certain other pixels, this means that these pixelsare not farther away from the certain other pixels than that maximumdistance.

In an embodiment of the method according to the present disclosure, thefirst motion only runs in a spatial direction orthogonal to an opticalaxis of the objective. However, in another embodiment of the method ofthe present disclosure, the first motion runs in two spatial directionsorthogonal to the optical axis of the objective. In the first motion,the sample may, for example, be brought out of a central position oneafter the other into eight further positions arranged around thiscentral position, particularly into positions of a square grid of rasterpoints, wherein the intensity of the light coming from the sample isregistered for each of these positions at each of the pixels of theimage sensors. This motion may also be repeated. Thus, nine intensitiesor a multiple of nine intensities are registered at each pixels of theimage sensor, and the variation of the intensities belonging to therespective pixel is determined from these intensities. In this way,pixels are selected that register object images of suitable referenceobjects for determining any lateral movements of the sample with respectto the objective.

If the first motion runs in a first direction or plane, in which themovements of the sample with respect to the objective are detected, thesample, during a second partial period of the set-up period, may be setin a second motion with respect to the objective, which runs in a seconddirection in which the movements of the sample with respect to theobject are detected, and which runs normal to the first direction orplane. In an embodiment, the second direction may be the direction ofthe optical axis of the objective. If second variations over the secondpartial period resulting from the second movement are determined andused in selecting the subset of the pixels of the image sensor as apositive criterion, pixels are selected by means of this positivecriterion, that correspond to object images of reference objects whichare well suited for tracking axial movements of the sample with respectto the objective.

In an embodiment, the subset of the pixels of the image sensor may beselected here such that, at its pixels or within a predetermined maximumsecond distance to its pixels, the second variations or the correctedsecond variations exceed a second variation limit value.

In a more particular embodiment, a second pre-selection for the subsetof the pixels may be made in that all pixels are selected at which thesecond variations or the corrected second variations exceed the secondvariation limit value and in that all pixels are added which are withinthe predetermined maximum second distance to the pixels at which thesecond variations or the corrected second variations exceed the secondvariation limit value. Afterwards, a unified pre-selection for thesubset of the pixels may be made in that a union of the firstpre-selection and the second pre-selection is determined, i.e. in thatall pixels are selected which are included within at least one of thesetwo pre-selections.

It is to be understood that, generally, even three pre-selections may bemade and unified in that the sample, over each of three differentpartial periods, is only moved in one of the three spatial directionswith respect to the objective, and the associated variations at theindividual pixels are determined. Vice versa, only a singlepre-selection for the subset may be based on a single two-dimensionallateral movement of the sample with respect to the objective, and it maybe assumed that this pre-selection is also suitable for tracking anaxial movement of the sample with respect to the objective.

The variations determined may also be used as a suitable negativecriterion in selecting the subset of the pixels of the image sensor.

For example, it is possible to not move the sample with respect to theobjective in a third partial period of the set-up period. Then, thirdvariations occurring over the third period can be determined and used inselecting the subset of the pixels of the image sensor as a negativecriterion. If variations of the intensities of the light registered atone of the pixels, even without movement of the sample with respect tothe objective, significantly go beyond variation background values whichare due to statistics, this may particularly have two causes, namely, onthe one hand, movements of structures within the sample or, moregeneral, parts of the sample inclusive of a sample slide or a coverslip, and, on the other hand, defective pixels of the image sensor. In asame way as defective pixels of the image sensor should not be used fordetecting movements of the sample with respect to the objective, pixelsshould not be used which capture structure images of structures movingwithin the sample. Thus, the third variations are a comprehensiblenegative criterion for the selection of the subset of the pixels.

In an embodiment, the subset of the pixels of the image sensor may beselected based on this negative criterion such that, at its pixels orwithin a predetermined maximum third distance to its pixels, the thirdvariations or the corrected third variations do not exceed a thirdvariation limit value. In a more particular embodiment, a thirdpre-selection for the subset of the pixels may be made in that pixelsare removed at which the third variation or the corrected thirdvariation exceeds the third variation limit value and in that, further,all pixels are removed which are within the predetermined maximum thirddistance to the pixels at which the third variations or the correctedthird variations exceed the third variation limit value. This thirdpre-selection may then be combined with the first pre-selection or, if asecond pre-selection has also been made, be combined with its union withthe first pre-selection in that an intersection of the thirdpre-selection and the first pre-selection or the union is determined.

In an alternative embodiment of applying the third variations as anegative criterion, an pixel weighting mask, whose transparencydecreases with increasing third variation or corrected third variationat the respective pixel, is applied to the images and the at least onereference image. This image mask may then, if present, also be appliedto the first pre-selection or the unified pre-selection of the pixels.

Besides pixels at which variations occur even without moving the samplewith respect to the object, also such pixels may not be considered ormay be removed in selecting the subset, which do not keep a fourthdistance to a margin of the image sensor. This fourth distance, in asame way as the previously mentioned first and second distances, has thefunction of ensuring that the object images of suitable referenceobjects are covered by the selected subset, even if the associatedreference objects move with respect to the objective together with thesample so that their images on the image sensor are moved. On the otherhand, the previously mentioned third distance has the function to avoidthat the structure images of structures moving within the sample getinto the area of the selected subset. Suitable sizes of the distancesdepend on the conditions of the imaging of the sample onto the imagesensor. Typically, the distances are in a range from 10 to 100 pixels,often in a range from 20 to 50 pixels.

Insofar as here and elsewhere ordinal numbers like “first”, “second”,“third” and “fourth” are used, these ordinal numbers only serve fordifferentiating the terms to which they are added. Thus, a thirdpre-selection for the subset of the pixel does not require that there isa first and/or second pre-selection of the subset of the pixels.Instead, only the pre-selection of the pixels may be made. Further, thethird pre-selection may be made chronologically before any also madefirst and/or second pre-selection. Further, it is not required that thefeatures provided with different ordinal numbers differ. Thus, thesecond variation limit value may differ from the first variation limitvalue, but it does not need to.

Further, with regard to the variation limit values, it may be remarkedthat they are suitably selected dependently on the totality of thevariations or corrected variations compared thereto. Thus, therespective variation limit value may, for example, be set such that itis only exceeded by 10% or 5% or 1% of the variations or correctedvariations at the individual pixels.

It has already been explained that the third variations which resultwithout motion of the sample with respect to the objective, may beapplied as a negative criterion for the selection of the subset by meansof an pixel weighting mask. The selection of the subset of the pixels ofthe image sensor may even completely be implemented by means of an imagemask which is applied to the images and the at least one reference imagein comparing the parts of the images to the parts of the respective onereference image. Prior to applying this image mask implementing theselection of the subset in form of transparent regions, edges betweenthe transparent regions and non-transparent regions of the image maskmay be smoothened. Then, the influence of pixels in the area of theedges continuously decreases. Thus, it is avoided that high differencesbetween the images and the reference images occur because some objectimages of objects in the sample, which have not been selected asreference objects, cross the edge of the image mask.

The subset of the pixels may fully automatically be selected by means ofthe method of the present disclosure, i.e. computer-implemented andwithout any contribution of a human user. The relative size of thesubset of the pixels which are automatically selected depends on thepredetermined distances and the criteria for selecting the variationlimit values. Typically it does not amount to more than 75%, often notto more than 50% and most times not to more than 25% of the pixels ofthe image sensor. A suitable selection of the subset many times includesat least 1% and often at least 5% and/or many times at least 20 andoften at least 200 of the pixels of the image sensor.

A microscope according to the present disclosure comprises an objective,a sample holder for positioning a sample, and a device for detectingmovements of the sample with respect to the objective. The devicecomprises an image sensor including an array of pixels, onto which thesample is imaged by means of the objective. The image sensor isconfigured to record images of the sample in that light coming from thesample is registered at the pixels of the image sensor. A selectionmodule of the device is configured to select a subset of not more than90% of the pixels of the image sensor. For this purpose, the selectionmodule is configured to determine variations of intensities of the lightcoming from the sample and registered at the individual pixels of theimage sensor during a set-up period, and to use the variations as acriterion in selecting the subset of the pixels of the image sensor. Acomparison module of the device is configured to compare the parts ofthe images, which each correspond to the selected subsets of the pixelsof the image sensor, to parts of at least one reference image, whichalso correspond to the selected subset of the pixels of the imagesensor. Movements of the sample with respect to the objective detectedas a result of this comparison may be used by a correction module tocontrol the sample holder for compensation movements which compensatethese movements. Thus, the sample is effectively kept at rest withrespect to the objective.

Corresponding preferred embodiments of the microscope of the presentdisclosure result from the preceding explanations of preferredembodiments of the method of the present disclosure.

In an alternative method of detecting movements of a sample with respectto an objective of the present disclosure, the sample is imaged onto animage sensor by means of the objective; images of the sample arerecorded by the image sensor in that light coming from the sample isregistered at the pixels of the image sensor; a subset of not more than90% of the pixels of the image sensor is selected in that, fora set-upperiod which is by at least 100%, preferably by at least 500%, even morepreferably by at least 1,000% longer than a duration of exposure of theimages and while the sample is not moved with respect to the objective,average intensities of the light coming from the sample and registeredat the individual pixels of the image sensor are determined, in thatvariations of the average intensities over groups of 9 to 625 pixelswhich are neighboring each other are determined, and in that thevariations or corrected variations which are corrected as above are usedas a positive criterion in selecting the subset of the pixels of theimage sensor. The parts of the images, which correspond to the selectedsubset of the pixels of the image sensor, are then compared to parts ofat least one reference image, which also correspond to the selectedsubset of the pixels of the image sensor. Actually, the averageintensities over the groups of pixels which are neighboring each othermay each be assigned to a central pixel of the respective group and thenbe used like in making the first pre-selection in the method of thepresent disclosure. Further, all embodiments of the previously describedmethod of the present disclosure which fit thereto are preferredembodiments of the alternative method of the present disclosure.

The alternative method locates areas of the images with strong spatialvariations of the intensities of the light coming from the sample whichare not levelled by movements of structures in the sample. However, thealternative method does not recognize if these strong variations are dueto optical artifacts or errors of pixels of the image sensor.

Referring now in greater detail to the drawings, the microscope 1depicted in FIG. 1 comprises an objective 2 and a sample holder 3 havingactuating elements for positioning a sample 4 with respect to theobjective 2, which are not depicted here in further detail. By means ofthe sample holder 3, the sample 4 can be positioned with respect to theobjective 2 laterally, i.e. in x- and y-direction, and axially indirection of an optical axis of the objective, i.e. in z-direction. Forimaging structures of interest of a sample by means of laser scanningfluorescence light microscopy, the microscope 1 has an excitation lightsource 5 for excitation light 6, a scanner 7 and a detector 8 forfluorescence light 9 coming from the sample 4. Further, a depletionlight source 10 for depletion light 11, for example STED-light, isprovided to increase the spatial resolution of the laser scanningfluorescence light microscopy. Here, a light distribution of thedepletion light 11 in the sample 4 is formed by means of a wave frontmodulator 12 such that it comprises a central intensity minimum. Insteadof STED-microscopy, the microscope 1 may, for example, also beconfigured for MINFLUX-microscopy. For this purpose, a light intensitydistribution of the excitation light 6 may be formed by an optionalfurther wave front modulator 12 such that it has a central intensityminimum. However, the microscope 1 does not need at all to be a laserscanning microscope but it may also be a localization microscope. In anycase, a device 13 for detecting movements of the sample 4 with respectto the objective 2 is present. The device 13 includes an image sensor 14comprising an array of pixels. The image sensor 14 is configured forrecording images of the sample in that light 29 coming from the sampleis registered at the pixels of the image sensor 14. This light 29 isgenerated by means of an illumination light source 15 for illuminationlight 28 by which the sample 4 is illuminated with incident light, here.In imaging the sample 4 onto the image sensor 14, a stop 16 is arrangedin a Fourier plane with respect to the image plane, the stop masking outa central area of the Fourier-plane and, correspondingly, low spatialfrequencies from the images recorded by the image sensor 14. The imagesrecorded by the image sensor 14 are processed in a processing unit 17 ofthe device 13. The processing unit 17 includes a selection module 18 forselecting a subset of pixels of the image sensor which are then used ina comparison module 19 as a basis for a comparison of the images toreference images. Movements of the sample 4 with respect to theobjective 2 detected on basis of this comparison are compensated bymeans of a correction module by controlling the sample holder 3. Thedevice 13 may completely correspond to that what is known from WO2020/201 430 A1 which is completely incorporated herein by reference.The imaging system for imaging the sample 4 on the image sensor 14 mayalso be taken from there. Thus, the positions of lenses 21-23 in thebeam path between the objective 2 and the image sensor 14 is notexplained in further detail here. The arrangement of beam splitters24-27 for separating or combining the individual beam paths in FIG. 1 isself-explanatory.

An embodiment of the method of the present disclosure is depicted inFIG. 2 as a flowchart. A first step 30, the imaging of the sample 4 ontothe image sensor 14, is realized by the optical setup of the microscope1 according to FIG. 1. The following steps 31 to 42 are executed by theselection module 18. In step 31, in a first partial period of a set-upperiod, the sample 4 is moved in a first direction with respect to theobjective 2. In step 32, first variations of the intensities of thelight 29 coming from the sample 4, which occur at the individual pixels,are determined. In step 33, a first pre-selection for the subset of thepixels to be considered by the comparison module 19 is made using thefirst variations or first variations corrected for purely statisticalinfluences as a positive criterion. Then, in step 34, the sample ismoved in a second direction. In step 35, the resulting second variationsof the light intensities of the light 19 from the sample at theindividual pixels of the image sensor 14 are determined. In step 36, asecond pre-selection for the subset of the first pixels is made usingthe second variations or second variations corrections for purelystatistical influences as a positive criterion. In step 37, the firstpre-selection from step 33 and the second pre-selection from step 36 arecombined in a unified pre-selection in that a union is determined. Instep 38, the determination of third variations takes place in a thirdpartial period of the set-up period without movements of the sample 4with respect to the objective 2. In step 39, a third pre-selection forthe subset is made on basis of these third variations or on basis ofthird variations corrected for statistical influences used as a negativecriterion. Then, in step 40, an intersection between the unifiedpre-selection from step 37 and the third pre-selection from step 39 isdetermined. In step 41, the pixels at the margins of the image sensor 14are removed from this intersection. Afterwards, in step 42, edgesbetween the areas of selected and non-selected pixels are smoothened. Animage mask resulting therefrom, in which the regions of selected pixelsare transparent and the regions of non-selected pixels arenon-transparent, is applied by the comparison module 19 in a step 43 incomparing the images with reference images. This comparison mayparticularly be made on basis of the determination of relations betweenthe respective image and a reference image. Details concerning thisaspect may be taken from WO 2020/201 430 A1.

FIG. 3A is a greyscale depiction of the average values of theintensities of an image recorded by the image sensor 14 according toFIG. 1 over the first partial period of the set-up period, in which thesample 4 is moved with respect to the objective 2. This movement may,for example, be implemented such that the sample, within the x-/y-planerunning orthogonally to the optical axis of the objective 2, is broughtone after the other into one of 3 x 3 neighboring positions with respectto the objective 2. In each of these positions, the intensity of thelight 29 from the sample 4 is registered. FIG. 3A shows the averagevalues of the corresponding nine intensity values.

On the other hand, FIG. 3B is a greyscale depiction of the standarddeviations of the intensities at the individual pixels of the imagesensor 14. FIG. 3C is a greyscale depiction of the standard deviationsaccording to FIG. 3B after their normalization as it will be explainedin further detail below. FIG. 3D is the result of a comparison of thedistribution of the normalized standard deviations according to FIG. 3Cwith a variation limit value whose determination will also be explainedbelow. FIG. 3E shows the result, if all points within a predeterminedmaximum distance are added to each of the pixels exceeding the variationlimit value according to FIG. 3D so that the object images of referenceobjects in the sample 4, which correspond to pixels above the variationlimit value according to FIG. 3D, are still found on the image sensor 14in one of the areas according to FIG. 3E even with a relative movementof the sample 4 with respect to the objective 2.

Whereas FIGS. 3A to 3E illustrate the determination of the firstpre-selection according to the steps 31-33 according to FIG. 2 in moredetail, FIGS. 4A-4E illustrate the determination of the thirdpre-selection according to the steps 38 and 39 in FIG. 2. Here, nomovement of the sample 4 with respect to the objective 2 occurs.Variations of the intensity of the light 29 from the sample 4 resultingat the pixels of the image sensor 4 are thus completely due tostatistical effects, movements of structures within the sample andfaulty pixels of the image sensor 14.

The distribution of the average intensities according to FIG. 4A doesnot noticeably differ from that one of the average intensities accordingto FIG. 3A. On the other hand, the distribution of the standarddeviations of the variations according to FIG. 4B is very different tothat one according to FIG. 3B. The differences of the distribution ofthe normalized standard deviations according to FIG. 4C to that oneaccording to FIG. 3C and the result of the comparison of the normalizedstandard deviations with a third variation limit value according to FIG.4D to the result of the comparison according to FIG. 3D are even higher.In FIG. 4E, the third pre-selection is depicted in such a way that theselected pixels are bright here, whereas the non-selected or deselectedpixels are black. The depiction therefore differs from that one of thefirst pre-selection in FIG. 3E.

Thus, in the intersection of the first and the third pre-selectionaccording to FIG. 5A the areas marked in black in FIG. 4E are missing ascompared to the first pre-selection according to FIG. 3E. According toFIG. 5B, as taking place in the step 41 of FIG. 2, pixels at the marginsof the image sensor 4 are additionally removed. According to FIG. 5C, astaking place in step 42 of FIG. 2, the edges of the subset of the pixelscorresponding to FIG. 5B are smoothened. An image mask corresponding toFIG. 5C may then be used in step 43 of FIG. 2 in comparing the images tothe reference images for detecting movements of the sample 4 withrespect to the objective 2.

FIG. 6A is a plot of the standard deviations of the intensities of thelight 29 from the sample 4 over the third partial period withoutmovement of the sample, as they have been determined in step 38 of FIG.2, over the respective average value of the intensities over the thirdpartial period. In other words, for each average value of theintensities or the root thereof depicted in FIG. 4A in a greyscale, theassociated standard deviation is plotted according to the greyscale inFIG. 4B. With a continuous line, FIG. 6A shows the result of a linearregression over all value pairs plotted. With a dashed line, the resultof a further linear regression for all value pairs below the continuousline is depicted. Thus, the dashed line indicates a variation backgroundvalue depending on the root extracted from the average intensity. FIG.6B shows the value pairs according to FIG. 6A after normalizing of therespective standard deviation to the variation background value. Thisnormalized standard deviations are depicted in FIG. 4C as greyscales.FIG. 6C is a histogram of the normalized standard deviations. Here, thefrequency of the standard deviations in different size classes of thenormalized standard deviation are plotted in an algorithmic way. By, forexample, determining a 99% percentile, the variation limit value fordetermining the significant normalized variations, which depends on therespective sample, may be determined. However, the significantnormalized variations shown in FIG. 4D are based on a variation limitvalue set to 3 based on common statistical significance criteria.

FIGS. 7A to 7C show the depictions of the first variations determinedduring the first partial period of the set-up period corresponding toFIGS. 6A-6C. However, no new variation background value which depends onthe square root drawn from the average intensity is determined from FIG.7A; instead that one according to FIG. 6A is used, i.e. the dashed linein FIG. 7A is the same one as in FIG. 6A. That the dashed line in FIG.7A appears to have a smaller slope is due to the fact that the standarddeviations of the intensities determined in the step 33 of FIG. 2 are,also relatively, clearly higher than those of the intensities determinedin step 38 of FIG. 2. FIG. 7B shows the value pairs according to FIG. 7Aafter normalizing the respective standard deviation to the deviationbackground value according to FIG. 6A. These normalized standarddeviations are depicted as greyscales in FIG. 5C. FIG. 7C is a histogramof the normalized standard deviations corresponding to FIG. 6C. Onceagain, by means of, for example, determining a 99% percentile, thevariation limit value for determining the significant normalizedvariations, which depends on the respective sample, may be determined.However, the significant normalized variations depicted in FIG. 5D arebased on a variation background value set to 10. This higher variationbackground value counts for the fact that the standard deviations of theintensities determined in the step 33 of FIG. 2 during moving thesample, even after their normalizing, are clearly higher than those ofthe intensities determined in the step 38 of FIG. 2 with the sample thatis not moved.

If special reference objects, like for example gold nanorods, whoseimages are well suited for detecting movements of the sample 4 withrespect to the objective 2 are included in the sample 4, the method ofthe present disclosure automatically selects the pixels belonging to theobject images of these artificial reference objects. In this case,structures of the sample are only rarely and only then selected if theyare similarly well suited as reference objects as the special referenceobjects. If, however, no special reference objects are introduced intothe sample, the method of the present disclosure—at least when using thepercentiles according to FIG. 7C—selects the structures of the samplethat are best suited as reference objects. All that occurs completelyautomatically. If, however, the normalized standard deviations above the99% percentile according to FIG. 7C, due to missing special referenceobjects and structures of the respective sample suitable as analternative, are only small and does not significantly exceed thenormalized standard deviations above the 99% percentile according toFIG. 6C, the method of the present disclosure—independently on any otheruse of the percentile according to FIG. 7C—may put out a warning messageindicating that the respective sample is not well suited for detectingits movement with respect to the objective.

Many variations and modifications may be made to the preferredembodiments of the invention without departing substantially from thespirit and principles of the invention. All such modifications andvariations are intended to be included herein within the scope of thepresent invention, as defined by the following claims.

I claim:
 1. A method of detecting movements of a sample with respect toan objective, the method comprising imaging the sample onto an imagesensor which comprises an array of pixels by means of the objective;recording images of the sample by the image sensor in that light comingfrom the sample is registered at the pixels of the image sensor;determining variations of intensities of the light coming from thesample and registered at the pixels of the image sensor during a set-upperiod in that a temporal course of the intensity of the light, whichhas been registered at a respective one of the pixels of the imagesensor over the set-up period, is analyzed; selecting a subset of notmore than 90% of the pixels of the image sensor using the variations asa criterion; and comparing parts of the images that each correspond tothe selected subset of the pixels of the image sensor to parts of atleast one reference image that also correspond to the subset of thepixels of the image sensor.
 2. The method of claim 1, wherein, in thestep of determining, the variations are determined in that a standarddeviation of the intensity of the light from an average intensity of thelight, which has been registered at the respective pixel of the imagesensor, is calculated.
 3. The method of claim 1, wherein, in the step ofdetermining, the sample, during a first partial period of the set-upperiod, is set in a first motion with respect to the objective, andfirst variations over the first partial period resulting from the firstmotion are determined, and wherein, in the step of selecting, the firstvariations are used as a positive criterion in selecting the subset ofthe pixels of the image sensor.
 4. The method of claim 3, wherein, inthe step of selecting, the subset of the pixels of the image sensor isselected such that the first variations exceed a first variation limitvalue at the pixels of the subset or within a predetermined maximumfirst distance to the pixels of the first subset.
 5. The method of claim4, wherein, in the step of selecting, a first pre-selection for thesubset of the pixels is made in that all pixels are selected at whichthe first variations exceed the first variation limit value and in thatall pixels are added which are located within the predetermined maximumfirst distance to the pixels at which the first variations exceed thefirst variation limit value.
 6. The method of claim 5, wherein, in thestep of determining, the first motion runs in a first direction or planein which the movements of the sample with respect to the objective aredetected, wherein, in the step of determining, the sample, during asecond partial period of the set-up period, is set in a second motionwith respect to the objective, that runs in a second direction in whichthe movements of the sample with respect to the objective are detectedand which is normal to the first direction or plane, and secondvariations resulting from the second motion over the second partialperiod are determined, and wherein, in the step of selecting, the secondvariations are used as a positive criterion in selecting the subset ofthe pixels of the image sensor.
 7. The method of claim 6, wherein, inthe step of selecting, the subset of the pixels of the image sensor isselected such that the second variations exceed a second variation limitvalue at the pixels of the subset or within a predetermined maximumsecond distance to the pixels of the subset.
 8. The method of claim 7,wherein, in the step of selecting, a second pre-selection for the subsetof the pixels is made in that all pixels are selected at which thesecond variations exceed the second variation limit value and in thatall pixels are added which are located within the predetermined maximumsecond distance to the pixels at which the second variations exceed thesecond variation limit value.
 9. The method of claim 8, wherein, in thestep of selecting, a unified pre-selection for the subset of the pixelsis made in that a union of the first pre-selection and the secondpre-selection is determined.
 10. The method of claim 5, wherein, in thestep of determining, the sample, during a third partial period of theset-up period, is not moved with respect to the objective, and thirdvariations occurring over the third partial period are determined, andwherein, in the step of selecting, the third variations are used as anegative criterion in selecting the subset of the pixels of the imagesensor.
 11. The method of claim 10, wherein, in the step of selecting,the subset of the pixels of the image sensor is selected such that thethird variations do not exceed a third variation limit value at thepixels of the subset or within a predetermined maximum third distance tothe pixels of the subset.
 12. The method of claim 11, wherein, in thestep of selecting, a third pre-selection for the subset of the pixels ismade in that all pixels are removed at which the third variations exceedthe third variation limit value and in that further all pixels areremoved which are within the predetermined maximum third distance to thepixels at which the third variations exceed the third variation limitvalue.
 13. The method of claim 12, wherein, in the step of selecting, acut-set of the third pre-selection and the first or unifiedpre-selection is determined in selecting the subset of the pixels of theimage sensor.
 14. The method of claim 10, wherein, in the step ofselecting, the third variations are used as a negative criterion inselecting the subset of the pixels of the image sensor in that an pixelweighting mask is applied to the images and the at least one referenceimage, whose transparency decreases with increasing third variation atthe respective pixel.
 15. The method of claim 1, wherein, in the step ofdetermining, the variations are adjusted with regard to at least one ofa background noise at the pixels of the image sensor in that from eachvariation a constant background noise value is subtracted, and astatistically caused variation background depending on a square root ofan average intensity of the light registered at the respective pixel ofthe image sensor, in that the variation background value at therespective pixel is subtracted from each variation or each variation isdivided by the variation background value at the respective pixel. 16.The method of claim 1, wherein, in the step of selecting, pixels whichdo not keep a fourth distance to a margin of the image sensor are notconsidered or removed.
 17. The method of claim 1, wherein, in the stepof comparing, an image mask is applied to the images and the at leastone reference image, that implements the selection of the subset of thepixels of the image sensor by transparent regions.
 18. The method ofclaim 17, wherein edges between the transparent regions andnon-transparent regions of the image mask are smoothened prior toapplying the image mask.
 19. The method of claim 1, wherein, in the stepof selecting, the subset is selected such as to not include more than75% or 50% or 25% of the pixels of the image sensor.
 20. A microscopecomprising an objective, a sample holder for positioning a samplerelative to the objective, and a device for detecting movements of thesample with respect to the objective, wherein the device comprises animage sensor including an array of pixels, onto which the sample isimaged by means of the objective, the image sensor being configured forrecording images of the sample in that light coming from the sample isregistered at the pixels of the image sensor, a selection moduleconfigured to determine variations of intensities of light coming fromthe sample and registered at the individual pixels of the image sensorduring a set-up period in that a temporal course of the intensity of thelight registered at the respective pixel of the image sensor over theset-up period is analyzed, and to use the variations as a criterion inselecting a subset of not more than 90% of the pixels of the imagesensor, and a comparison module configured to compare parts of theimages which each correspond to the subset of the pixels of the imagesensor with parts of at least one reference image which also correspondto the subset of the pixels of the image sensor.